Computer based systems such as pen based systems and "electronic whiteboards", provide graphical user interfaces based on utilization of an input device such as a pen, stylus or a cursor control device. Such systems are often a combination of hardware and software and typically enable freeform input and are termed freeform graphics systems. In such systems the "display" shows a workspace that acts as both a means for showing data and as an input medium. Graphic objects may be "drawn" on the display via pen strokes created using the input device. The graphic objects may represent words, figures or anything that can be displayed. Various operations can then be performed on the graphic objects using an input technique known as gestures. Gestures are themselves merely pen strokes that are interpreted as instructions. Sometimes, such systems will have a draw mode of operation and a gesture mode of operation to distinguish when a pen stroke creates a persistent graphic object or when the pen stroke is treated as a gesture.
To facilitate user interactions, the LiveBoard system provides the notion of structures. Various types of structures utilized by the LiveBoard system are described in EPA 0 667 567 A2, entitled "Apparatus and Method For Supporting the Implicit Structure of Freeform Lists, Outlines, Text, Tables, and Diagrams In A Gesture-Based Input System and Editing System", which is assigned to the same assignee of the present application. Conceptually a structure is a collection of graphic objects that have a particular relationship, e.g. a list of items written on the LiveBoard. When a structure operation occurs the relationship amongst the items is retained. For example, when an item is inserted into the middle of a list, the items below the insertion point are moved down in order to make room for the inserted item.
When using such freeform graphics systems as the LiveBoard, it has been determined that often the information being worked on is structured data. Structured data is a representation of information wherein the individual data components form a defined relationship. For example, the data found in individual cells of a spreadsheet application comprise the spreadsheet information. As the "value" contained in one cell may depend on the values in other cells, a structure relationship is formed.
As known freeform editing programs do not look at the underlying content of graphic objects, they do not facilitate operation wherein the content of such graphic objects have such relationships. On the other hand, a structured application is usually too rigid, constraining the use to the confines of predefined structures.
It would be desirable to flexibly combine the best of both kinds of applications--some facilities to help with the structured data, but the freedom to annotate or make marks of any kinds when needed.
One approach to doing this would be to embed Object Link Embedding (OLE) objects into a freeform editing program. OLE is functionality provided in the Windows.TM. Operating System, which enables application objects to be linked or embedded into other applications. This approach is found in a product called the Meeting Board, which was available from LiveWorks, an affiliated company of the Xerox Corporation. But this approach is awkward, since each object evokes its own application, leading to a multiplicity of different user interfaces on the display. Also, the elements of the OLE objects are not integrated with the other objects of the freeform editing program; e.g., the individual cells of an embedded spreadsheet are not accessible to the freeform editing program.
Other related technology includes the ActiveX Data Objects and related programs available from the Microsoft Corporation.